High resolution perception of content in a wide field of view of a head-mounted display

ABSTRACT

Introduced herein are various techniques for displaying virtual and augmented reality content via a head-mounted display (HMD). The techniques can be used to improve the effectiveness of the HMD, as well as the general experience and comfort of users of the HMD. An HMD may increase and/or decrease the resolution of certain areas in digital content that is being viewed to more accurately mimic a user&#39;s high resolution and low resolution fields of view. For example, the HMD may monitor the user&#39;s eye movement to identify a focal point of the user&#39;s gaze, and then increase the resolution in an area surrounding the focal point, decrease the resolution elsewhere, or both. Predictive algorithms could also be employed to identify which areas are likely to be the subject of the user&#39;s gaze in the future, which allows the HMD to present the regionally-focused content in real-time.

CROSS REFERENCE TO RELATED APPLICATION

This application claims benefit of U.S. Provisional Patent ApplicationNo. 62/077,100, filed Nov. 7, 2015 and titled “Methods and Systems forHead Mounted Display,” which is incorporated by reference herein in itsentirety.

RELATED FIELD

The present technology relates to display devices for virtual andaugmented reality content, and more specifically to methods and systemsthat improve the effectiveness of head-mounted displays (HMDs) and thecomfort of users.

BACKGROUND

HMDs, such as helmet-mounted displays and eyeglass-mounted displays, aredisplay devices, worn on the head (e.g., as part of a helmet,eyeglasses, or visor) that has a display optic in front of one(monocular HMD) or both eyes (binocular HMD). As further describedbelow, an HMD typically includes optical displays with lenses andsemi-transparent mirrors. The displays are generally miniaturized andmay include light emitting diodes (LEDs), liquid crystal displays(LCDs), etc.

HMDs differ from one another in several aspects. For example, some HMDscan only display a computer-generated image (CGI), which may be referredto as a virtual image. Other HMDs are able to show a combination of aCGI and live images from the real world. Such embodiments can bereferred to as augmented reality.

Although HMDs have been developed for use in numerous applications(e.g., gaming, medicine, training/simulation), continued use oftencauses the user's eyesight to be strained. Extended exposure to virtualand augmented reality environments has also been associated withsymptoms similar to motion sickness, such as general discomfort,headache, nausea, and disorientation. Therefore, a need exists toimprove the general effectiveness of HMDs and the comfort of users,especially those who are exposed to virtual or augmented reality forextended periods of time.

DISCLOSURE OVERVIEW

Introduced herein are a variety of embodiments for methods and systemsfor displaying virtual and augmented reality content to a user throughan HMD. Various embodiments described herein can also be used to improvethe effectiveness of an HMD and the comfort and general experience of auser of the HMD. For example, a binocular HMD system can assist users bymaintaining and merging two separate images comfortably without visualdiscomfort. The HMD system could, for example, present a visualstabilizer to each eye that can be merged together in binocular view tocreate a strong fusional system. A strong fusional system often resultsin a decrease in eye fatigue and strain experienced by the user, therebycreating a binocular HMD system and improving the user experience.

An HMD system could also improve the visual comfort and binocularstability of the user by modifying the size of the overlap (i.e., theportion of two distinct images that can be seen simultaneously by eachof the user's eyes). The HMD system can adjust the overlap of twoseparate images presented to each of the user's eyes based on, forexample, the perceived focal distance and type of application. In someembodiments, these adjustments occur in real-time.

The HMD system may also provide improved resolution perception in a widefield of view by decreasing the resolution of an image (i.e., a CGI)viewed by a user in a first area and/or increasing the resolution in asecond area based on the user characteristics. Sample characteristicsinclude the speed of the user's head movement and the motion of theuser's eyes. The HMD system can also include a tracking system thataccurately measures the speed of the user's head movement and the motionof the user's eyes. These measurements can be used to dramaticallyimprove the quality, perception, and stability of digital contentdisplayed to a user.

Although some existing HMDs employ three-dimensional (“3D”) depthsensing systems for tracking and detecting the user's hand motions andgestures, the resolution and accuracy of these systems are poor outsideof a limited number of lighting environments (e.g., indoor lighting).Various embodiments described herein, however, can modify and/or adapt auser interface and/or an HMD input system based on the illuminance asmeasured by one or more sensors coupled to the HMD. Several HMD systemsdescribed herein improve the resolution and accuracy of 3D depth sensingsystems in a broad range of lighting environments (e.g., bright,outdoor).

In some embodiments, a combined HMD system includes a first opticconfigured to present digital content up to a predetermined distanceaway (e.g., 0.5, 1, 2.5 meters) and a second optic configured to presentdigital content whose distance exceeds the predetermined distance. Forexample, the first optic may be adapted for close-range viewing, whilethe second optic may be adapted for long-range viewing. Traditionally,binocular HMD systems have been designed for either long-range orclose-range viewing. That is, binocular HMD systems typically have twoseparate displays and two separate optical systems for presentinglong-range and close-range content to a user. The various HMD systemsdescribed herein can further include a visual balancing system (e.g.,managed by visual balancing software) that is able to properly balancethe different images and improve the user's viewing experience.

An HMD could also be configured to track gestures and/or vocal commandsof the user. Gestures may include, for example, pushing the nose bridgeof the frame of the HMD, applying pressure to the frame with one or bothhands, and pushing the frame upward or laterally. The gestures and vocalcommands can be used to control and/or modify the user interface of theHMD, trigger a command input (e.g., select, zoom, modify overlap, modifyresolution), or some combination thereof.

Various embodiments of the HMD system described herein can alsoimplement animated digital content that simulates the change of distanceas observed by the user. Effective superimposition of digital contentrequires constant measuring, processing, and analyzing of informationrelated to distance of the observed focal object, calibration of acamera, eye gaze position, etc. Various properties (e.g., convergence,brightness) of the animated digital content can be modified to moreaccurately imitate the user's perceived change of distance.Superimposition techniques such as these are likely to be most usefulwhen employed by HMDs that present augmented reality content.

The Overview is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. Some embodiments have other aspects, elements, features,and steps in addition to or in place of what is described above. Thesepotential additions and replacements are described throughout the restof the specification. Other advantages and features will become apparentwhen viewed in light of the Detailed Description and when taken inconjunction with the attached drawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other objects, features and characteristics of the presentinvention will become more apparent to those skilled in the art from astudy of the following Detailed Description in conjunction with theappended claims and drawings, all of which form a part of thisspecification.

FIG. 1 is a schematic diagram showing basic components of an HMD.

FIG. 2A is a front view representation of a binocular HMD according tovarious embodiments.

FIG. 2B is a front view representation of a monocular HMD according tovarious embodiments.

FIGS. 3A-B are side view representations of HMDs according to variousembodiments.

FIG. 4 is a diagram illustration of a left eye view, right eye view, andcombined view showing overlap experienced by a user.

FIG. 5 is an inside view of an HMD that includes configurable icons.

FIGS. 6A-C depict how visual stabilizers can be used to readily andeffectively merge distinct digital images into a single composite image.

FIG. 7 is a diagram illustration of a user's high resolution and lowresolution viewing fields.

FIG. 8 depicts a user's high resolution and low resolution viewingfields in a single instance.

FIG. 9 is an illustration of high resolution and low resolution viewingareas in a digital image presented by an HMD.

FIG. 10 is a diagram illustration of a method for displaying digitalcontent over a range of focal distances.

FIG. 11 is a flowchart of a process for adjusting display resolution bythe HMD as may occur in some embodiments.

FIG. 12 is a flowchart of a process for modifying an interface and/orupdating the controls for an input management system as may occur insome embodiments.

FIG. 13 depicts retinal correspondence and non-correspondence points.

FIG. 14 shows a single focal point and the corresponding region ofsingle vision.

FIG. 15 is a block diagram illustration an example of a processingsystem in which at least some operations described herein can beimplemented, consistent with various embodiments.

The figures depict various embodiments of this disclosure for purposesof illustration only. One skilled in the art will readily recognize fromthe following Detailed Description that alternative embodiments of thesystems and methods illustrated herein may be employed without departingfrom the principles of the invention described herein.

DETAILED DESCRIPTION

References will be made below in the Detailed Description to variousembodiments, which are illustrated in the accompanying drawings. Thesame or similar reference numerals have been used throughout thedrawings to refer to the same or like parts. The accompanying figuresare included to provide a further understanding of the invention. Itwill be understood by one skilled in the art that various features ofthe embodiments described within the Detailed Description and thefigures can be used in any and all combinations.

Terminology

Brief definitions of terms, abbreviations, and phrases used throughoutthis application are given below.

Reference in this specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least one embodimentof the disclosure. The appearances of the phrase “in one embodiment” invarious places in the specification are not necessarily all referring tothe same embodiment, nor are separate or alternative embodimentsmutually exclusive of other embodiments. Moreover, various features aredescribed that may be exhibited by some embodiments and not by others.Similarly, various requirements are described that may be requirementsfor some embodiments but not others.

Unless the context clearly requires otherwise, throughout thedescription and the claims, the words “comprise,” “comprising,” and thelike are to be construed in an inclusive sense, as opposed to anexclusive or exhaustive sense; that is to say, in the sense of“including, but not limited to.” As used herein, the terms “connected,”“coupled,” or any variant thereof, means any connection or coupling,either direct or indirect, between two or more elements; the coupling orconnection between the elements can be physical, logical, or acombination thereof. For example, two devices may be coupled directly,or via one or more intermediary channels or devices. As another example,devices may be coupled in such a way that information can be passedthere between, while not sharing any physical connection with oneanother. The words “associate with,” meanwhile, means connecting orrelating objects, items, etc. Additionally, the words “herein,” “above,”“below,” and words of similar import, when used in this application,shall refer to this application as a whole and not to any particularportions of this application. Where the context permits, words in theDetailed Description using the singular or plural number may alsoinclude the plural or singular number respectively. The word “or,” inreference to a list of two or more items, covers all of the followinginterpretations of the word: any of the items in the list, all of theitems in the list, and any combination of the items in the list.

If the specification states a component or feature “may,” “can,”“could,” or “might” be included or have a characteristic, thatparticular component or feature is not required to be included or havethe characteristic.

The term “module” refers broadly to software, hardware, or firmware (orany combination thereof) components. Modules are typically functionalcomponents that can generate useful data or another output usingspecified input(s). A module may or may not be self-contained. Anapplication or software program may include one or more modules, or amodule may include one or more applications or software programs.

The terminology used in the Detailed Description is intended to beinterpreted in its broadest reasonable manner, even though it is beingused in conjunction with certain examples. The terms used in thisspecification generally have their ordinary meanings in the art, withinthe context of the disclosure, and in the specific context where eachterm is used. For convenience, certain terms may be highlighted, forexample using capitalization, italics, and/or quotation marks. The useof highlighting has no influence on the scope and meaning of a term; thescope and meaning of a term is the same, in the same context, whether ornot it is highlighted. It will be appreciated that the same element canbe described in more than one way.

Consequently, alternative language and synonyms may be used for any oneor more of the terms discussed herein, and special significance is notto be placed upon whether or not a term is elaborated or discussedherein. Synonyms for certain terms are provided. A recital of one ormore synonyms does not exclude the use of other synonyms. The use ofexamples anywhere in this specification including examples of any termsdiscussed herein is illustrative only, and is not intended to furtherlimit the scope and meaning of the disclosure or of any exemplifiedterm. Likewise, the disclosure is not limited to various embodimentsgiven in this specification.

Ocular Operations and Implications

FIG. 13 depicts retinal correspondence and non-correspondence points.When the user's eyes 1302 focus on a focal point (e.g., retinalcorrespondence point 1380), three distinct viewing areas are naturallycreated: an area seen only by the user's right eye 1320, an area seenonly by the user's left eye 1322, and an area seen by both of the user'seyes 1324. The area seen by both of the user's eyes 1324 is alsoreferred to as the overlap.

In some embodiments, the user triggers one or more actions (e.g.,overlap modifications, user interface modification, brightness change)by focusing on a retinal correspondence point 1380. When a user focuseson a retinal correspondence point, each of the user's eyes 1302 share acommon subjective visual direction. More specifically, simultaneousstimulation results in the subjective sensation that the focal pointcomes from the same direction in space. If the simultaneous stimulationof the retinal areas in each of the user's eyes 1302 results in thesensation of two distinct visual directions for a target (i.e.,diplopia), the focal point is said to be a non-correspondence point1382.

Various techniques described herein can be implemented by observing thelocation and occurrence of the user's retinal correspondence points1380. For example, overlap modifications, placement of visualstabilizers, and augmented reality animations could all require (orprefer) that accurate tracking of the user's retinal correspondencepoints 1380 be performed. Accurate tracking of the user's retinalcorrespondence points 1380 may result in a more effective viewingexperience (e.g., better resolution, less eye fatigue) for the user.

FIG. 14 shows a single focal point 1480 and the corresponding region ofsingle vision. When each of the user's eyes 1402 fixate on a focal point1480 and normal retinal correspondence exists, the focal point is seensingly. That is, only a single image of the focal point is recognized bythe user. Normal retinal correspondence requires that the correspondingretinal areas in the user's two eyes 1402 bear identical relationshipsto the fovea in each eye (i.e., both corresponding areas are locatedequidistantly to the right/left and above/below the fovea). Pointslocated on either side of the focal point 1480 also fall on retinalcorrespondence points and can be seen singly. The points that can beseen singly constitute a horizontal construct known as the Vieth-Müllercircle. The Vieth-Müller circle includes a 3D curved surface, alsoreferred to as an empirical horopter.

Points not lying within the Vieth-Müller circle fall on retinalnon-correspondence points and would be expected to cause the user to seedouble (i.e., diplopia). However, double vision does not occur within alimited area surrounding the Vieth-Müller circle because the user'sbrain fuses the two distinct images captured by each of the user's eyes1402. This space, illustrated as a band in FIG. 14 and labeled “Regionof Single Vision,” is called Panum's area of single binocular vision.Objects outside Panum's area fall on retinal non-correspondence pointsand are seen as two distinct visual images (i.e., diplopia).

HMD systems can account for these phenomena by segmenting the user'sviewing field into at least three distinct fields. For example, a firstzone (e.g., Zone 1 of FIG. 14) may include digital content to bedisplayed up to 70 cm from the user. A second zone (e.g., Zone 2 of FIG.14) may include any digital content that is to be displayed between 70cm and 1.5 m. A third zone (e.g., Zone 3 of FIG. 14) may include digitalcontent to be displayed at a distance that exceeds 1.5 m. One skilled inthe art will recognize the fields described herein may correspond todifferent viewing distances and may increase or decrease in number. Theexample numbers used here are for illustration only. An increase in thenumber of viewing fields implemented by the HMD may result in a moreeffective display of digital augmented reality content.

The distinct viewing fields can be used by the HMD to trigger variousactions or modifications. In some embodiments, animated digital contentcan be configured to animate based on which viewing field includes thefocal point 1480. For example, if the HMD determines the focal point1480 is located in Zone 1, the HMD may project digital content thatcompletes a predetermined animation (e.g., slides forward toward theuser or shakes). Similarly, if the HMD determines the focal point 1480is located in Zone 2, the HMD system may project digital content thatcompletes another predetermined animation (e.g., slides away from theuser, decreases in brightness).

The HMD could also determine where to present augmented reality contentbased on input from the user. For example, the HMD may determine whichzone includes the focal point 1480 based on recognizing a particulargesture (e.g., pointing). Upon recognizing the gesture, the HMD mayperform an action (e.g., zoom to the focal point and present augmentedreality content).

In some embodiments, the overlap viewed by the user (e.g., overlap 1324of FIG. 13) is modified based on which viewing field includes the focalpoint 1480. For instance, if the HMD determines the focal point 1480 islocated in Zone 1, the HMD may project a larger overlap (i.e., more ofthe digital content is shared between the two images and can be seen byboth of the user's eyes). If the HMD instead determines the focal point1480 is located in Zone 2, the HMD may project a smaller overlap (i.e.,less content within the two images can be seen by both of the user'seyes).

Visual stabilizers embedded within the digital content can also bemodified based on which viewing field includes the focal point 1480. Forexample, if the HMD determines the focal point 1480 is located in Zone1, the HMD may generate and display a series of visual stabilizers(e.g., frame, clouds, trees). Alternatively, if the HMD determines thefocal point 1480 is located in Zone 2, the HMD may generate and displayanother series of visual stabilizers (e.g., frame and clouds only) thatdiffers from the original series of visual stabilizers. Variousproperties of the visual stabilizers (e.g., brightness, contrast, size)can be modified based on which viewing field includes the focal point1480.

The HMD can also include a tracking system that can accurately measurethe focal point of the user's eyes. The tracking system may include oneor more sensors configured to measure the speed and direction of theuser's head and eye movement. The sensor(s) could include, for example,motion/movement sensors (e.g., accelerometers) and/or optical sensors(e.g., infrared sensors). In some embodiments, the focal point of theuser's eyes is tracked in real-time. When the focal point is tracked inreal-time, the overlap may be modified and/or the digital content may beanimated in real-time.

System Overview

FIG. 1 shows several basic elements of an HMD, which includes an imagedisplay system 110 and an optical display surface 112 configured toreflect light 116. More specifically, light 116 can be emitted from theimage display system 110 and reflected by optical display surface 112towards the user's eye 102.

As described above, HMDs can be configured to display two differenttypes of digital imagery. First, some HMDs are configured to displaysimulated (e.g., computer-generated) digital images that together forman entirely virtual environment. This is referred to as “virtualreality.” Virtual reality generally requires the user wear a helmet,goggles, etc. that form an enclosed area around the user's eyes andblock out the real-world. Second, some HMDs are configured to displaysimulated (e.g., computer-generated) images that are integrated intoreal world content perceived by the user. This is referred to as“augmented reality.” Augmented reality allows the user to concurrentlyview virtual and real world imagery and, in some instances, interactwith virtual content that relates to what is seen the real world.Generally, the user is able to distinguish between the two. For example,augmentation may take place through superposition of a simulated imageon a non-simulated, real world image, which can allow the user to viewadditional data relevant to a desired task, the local environment, etc.In various embodiments, the simulated image can be adapted to allow foruser interaction.

HMD systems also fall into three general categories: binocular,bi-ocular, and monocular systems. Binocular HMD systems present aseparate image to each eye of the user, while bi-ocular HMD systemspresent a single image to both of the user's eyes, and monocular HMDsystems present a single image to only one eye of the user. See, forexample, FIGS. 2A-B, which depict a binocular HMD system and a monocularHMD system, respectively.

Because binocular HMD systems present distinct images to each of theuser's eyes, the user must make a constant fusional effort to maintain asimultaneous view of the content. The constant fusional effort requiredby binocular HMD systems causes eye fatigue and strain over time, one ofthe main causes of visual discomfort for users of augmented and virtualreality systems. Those users with weak binocular vision or preexistingvisual disorders are more likely to experience severe symptoms and/oraggravation of the preexisting visual disorder.

The content (e.g., simulated images) of binocular HMD systems isgenerally presented to the user with an overlap (i.e., the portion ofeach of the two separate images can be seen simultaneously by each ofthe user's eyes). The size of the overlap (illustrated in FIG. 4) mayvary based on the device type, content, application, etc.

One of the major challenges of augmented reality is correctlysuperimposing digital content onto the 3D real world environmentperceived by the user. Effective and efficient superposition iscomplicated due to the difficulty of accurately tracking the user'seyes, which constantly change their fixation distance on variousobjects.

FIG. 2A is a front view representation of a binocular HMD system 200Aaccording to various embodiments. The binocular HMD system 200A includesa frame 206, a bridge 208, an image display system 210, optical displaysurfaces 212L, 212R, and one or more sensors 214. The frame 206 can beconfigured to support the binocular HMD system 200A similar toconventional glasses. That is, the frame 206 may be supported by theuser's ears and the bridge 208 may be supported by the user's nose.Although the binocular HMD system 200A resembles a pair of conventionaleyeglasses, the binocular HMD system 200A could also resemble goggles, ahelmet, a visor, etc. The frame 206 of the binocular HMD system 200A canbe modified so long as the user 202 can view virtual or augmentedreality content on the optical display surfaces 212L, 212R (collectivelyreferred to as “optical display surface 212”) in the proper orientation.

The binocular HMD system 200A includes at least one image display system210 and at least one optical display surface 212. The optical displaysurface could include an optical display surface for the user's left eye212L and/or an optical display surface for the user's right eye 212R. Insome embodiments, a single optical display surface 212 can be viewed byboth eyes concurrently. The optical display surface 212 may also becurved in order to enlarge the viewing field and improve the viewingexperience of the user. In some embodiments the optical display surface212 is opaque, thereby allowing the user 202 to view virtual realitycontent without interruption, while in other embodiments the opticaldisplay surface 212 may be transparent or semi-transparent, therebyallowing the user 202 to view augmented reality and real world contentsimultaneously.

The optical display surface 212 may completely surround one or both eyesof the user. As described above, in some embodiments the optical displaysurface can comprise two separate optical display surfaces (e.g., 212Land 212R). The user's right eye may be shown a first representation of a3D digital image on optical display surface 212R, and the user's lefteye may be shown a second representation of the 3D digital image onoptical display surface 212L. The distinct optical display surfaces212L, 212R can be optimized to present the digital environment relativeto the location of the user's eyes. In such embodiments, the distinctoptical display surfaces 212L, 212R could be optimized independently orcollectively. In some embodiments, the optical display surface 212includes a single optical display surface, some portion of which isviewable by both eyes and some portion of which is viewable by only oneeye. The bridge 208 may have one or more reflectors arranged adjacent tothe optical display surface(s) 212 in order to provide an immersive andrealistic digital environment.

The binocular HMD system 200A can also include an electronics module218, which can process digital content, analyze data collected by thesensor(s) 214, optimize digital content presented to the user 202, etc.The electronics module 218 and binocular HMD system 200A may be poweredthrough a wired or wireless (e.g., battery) medium.

As further described below, the image display system(s) 210 can bepositioned in several different orientations. For example, the imagedisplay system(s) 210 could be coupled to the frame 206 and orientedtoward the optical display surface 212 or, alternatively, within (orbehind) the optical display surface 212 itself. In some embodiments, afirst image display system is configured to project light toward (i.e.,display digital images on) optical display surface 212L, and a secondimage display system is configured to project light toward opticaldisplay surface 212R. The image display systems could project the samedigital image, similar digital images, or different digital images tothe user 202. The placement of the image display system 210 may relateto the placement and angle of the optical display surface 212. Moreover,the placement and angle of image display system 210 may depend on one ormore properties (e.g., pixel count, resolution, brightness) of thedigital content that is to be projected onto, and reflected from, theoptical display surface 212. The image display system 210 may, forexample, use light emitting diodes (LEDs), liquid crystal displays(LCDs), organic light emitting diodes (OLEDs), or some combinationthereof. The image display system 210 generally uses small-form displayscapable of high resolution (i.e., high pixel density) display, which aremore readily able to create an immersive and realistic virtual realityenvironment for the user 202.

Sensor(s) 214 may be coupled to the frame 206 and/or the optical displaysurface 212 to monitor various aspects of the user's local real worldenvironment. For example, the sensor(s) could be coupled to oppositesides of the binocular HMD system 200A and configured to gather data,which is then processed and analyzed by the electronics module 218. Thedata gathered by the sensor(s) 214 can be used (e.g., by the electronicsmodule 218) to optimize the digital content that is presented to theuser 202. The sensor 214 is typically coupled to the electronics module218 to receive power. However, the sensor 218 could also be configuredto receive power from a distinct power source. Sensor(s) 214 may be acamera configured to capture the user's interactions with the localenvironment, a light sensor configured to track illuminance levelswithin the local environment, an audio sensor configured to sense vocalcommands, etc. In some embodiments, a plurality of sensors 214 arecoupled to the frame 208. The plurality of sensors 214 may be aplurality of the same sensor (e.g., multiple cameras or microphones) ora combination of different sensors, such as those described above. Theplurality of sensors 214 might be chosen such that a particular HMDsystem is adapted for particular applications (e.g., outdoor use,athletic activities, medical applications).

FIG. 2B is a front view representation of a monocular HMD system 200Baccording to various embodiments described herein. The monocular HMDsystem 200B includes a frame 206, a bridge 208, an image display system210, an optical display surface 212R, and one or more sensors 214. Themonocular HMD system 200B is, in large part, comparable to the binocularHMD system 200A described above. One skilled in the art will recognizethe similarities and, accordingly, only the differences will bediscussed.

Monocular HMD system 200B includes an image display system 210 and anoptical display surface 212R. In contrast to the binocular HMD system200A described above, monocular HMD system 200B has a single opticaldisplay surface 212 that is viewable by one of the user's eyes. See, forexample, FIG. 2B, which illustrates monocular HMD system 200B asincluding an optical display surface 212R for the user's right eye.Note, however, that the optical display surface 212 of the monocular HMDsystem 200B could be adapted for a user's left eye (e.g., opticaldisplay surface 212L) or right eye (e.g., optical display surface 212R).

In some embodiments, a binocular HMD system 200A actually includes twodistinct monocular optical displays. Thus, the binocular HMD system 200Acan include a first optical display configured to present augmented orvirtual reality content up to a predetermined distance away, and asecond optical display configured to present augmented or virtualreality content whose distance exceeds the predetermined distance. Forexample, the binocular HMD system 200A may include a first optic displaythat presents content for long-range viewing (e.g., greater than 1, 2.5,or 5 meters) and a second optic display that presents content forclose-range viewing (e.g., less than 1, 2.5, or 5 meters), therebycreating a combined HMD system. The first optic display may use opticaldisplay surface 212L, while the second optic display may use opticaldisplay surface 212R.

The combined HMD system may also include an electronics module (e.g.,electronics module 218) that executes visual balancing software or iscommunicatively coupled to a visual balancing module. The visualbalancing module may, for example, use a fusional stabilizer system tobalance the different images displayed on the first optical display andthe second optical display. In some embodiments, the combined HMD systemcan switch from a binocular viewing mode (i.e., first optical displayand second optical display displayed simultaneously) to a monocularviewing mode, whereby one image display system 210 (or part of the imagedisplay system 210) and its corresponding optical display surface (e.g.,212L) are turned off or put into sleep/hibernate mode. When the HMDsystem is in monocular viewing mode, digital content can only bepresented to one of the user's eyes.

In some embodiments, the combined HMD system may use progressive opticallenses for one or both of the user's eyes. Progressive optical lensesare characterized by a gradient of increasing lens power distributedthroughout the lens associated with each of the user's eyes.Accordingly, the first optical display and second optical display may beconfigured to use similar or different gradients depending on therequirements of each of the user's eyes. Progressive optic lenses couldalso be implemented in binocular HMD systems, thereby allowing the userto train both eyes on a single optical focal point.

FIGS. 3A-B are side view representations of HMDs (e.g., HMD systems200A, 200B of FIGS. 2A-B) according to various embodiments. An HMDsystem 300A can include a frame 306, an image display system 310, anoptical surface display 312, and an electronics module 318. The frame306 can be configured to be supported by the user's ears, while thebridge may be supported by the user's nose. In some embodiments, theimage display system 310 emits light 304 that is reflected by an opticaldisplay surface 312 toward the user's eye(s). Thus, the light 304 (whichforms a digital image) is reflected towards a user's eye(s) due to theprojection of the image display system 310 towards an angled opticaldisplay surface 312. HMD system 300A may have a binocular or a monocularoptical display configuration as described above. The electronics module318 can control what digital content is projected by the image displaysystem 310.

Similarly, HMD system 300B can include a frame 306, an image displaysystem 310, an electronics module 318, and, optionally, an opticalsurface display 312. In some embodiments, the image display system 310is configured to emit light 304 directly into the user's eye(s). Whenthe light 304 is emitted toward the user's eye(s), the optical displaysurface 312 may be present only in select embodiments. The opticaldisplay surface 312 may be included in order to improve image quality incertain environments or for certain users, but may not be necessarydepending on the arrangement of the image display system 310 and theoptical display surface 312. The optical display surface 312 may not berequired in some embodiments to redirect the light 304 emitted by theimage display system 310. As described above, HMD system 300B may have abinocular or monocular optical display configuration. An electronicsmodule 318 can process the digital content (e.g., virtual realitycontent, augmented reality content) projected by the image displaysystem 310 to one or both of the user's eyes.

The image display system 310 of HMD systems 300A and 300B may beconfigured to project virtual reality content, augmented realitycontent, or both. In various embodiments, the optical display surface312 is completely opaque, completely transparent, or semi-transparentdepending on the type of digital content to be displayed, the intendedviewing environment, etc. For example, the optical display surface 312may be opaque for virtual reality applications, and transparent orsemi-transparent for augmented reality applications. The optical displaysurface 312 could also be selected based on its reflective and/ortransmissive properties.

Adjustable Overlap

FIG. 4 is a diagram illustration of a left eye view, a right eye view,and combined view showing overlap experienced by a user 402. Morespecifically, FIG. 4 shows a user's field of vision broken into threecategories: the area seen only by the user's right eye 420, the areaseen only by the user's left eye 422, and the area seen by both of theuser's eyes 424. The overlap is the portion of the left eye view and theright eye view that can be viewed simultaneously by both eyes of theuser. Digital content displayed to the user of a binocular or combinedHMD system, such as binocular HMD system 200A of FIG. 2A, can bepresented with an overlap. Conventionally, the amount of overlap ispredefined or restricted by the hardware components of the HMD.Embodiments described herein, however, incorporate hardware componentsand/or software modules that allow the overlap to be adjusted inreal-time as digital content is being viewed by the user. The size ofthe overlap may vary based on a number of factors, including, but notlimited to, system specifications, digital content properties, contenttype, application, hardware properties, and software properties (e.g.,visual balancing parameters). In some embodiments, very precisemodifications of the overlap can be made (e.g., on a pixel-by-pixelbasis).

Overlap can vary based on several factors, including the perceived focaldistance, the content type, and the application type. For example,close-range viewing may require a larger overlap, while long-rangeviewing may require a smaller overlap. HMD systems with small overlapgenerally require more visual effort by the user to fuse the twodistinct images together.

In some embodiments, an HMD system is able to adjust the overlap of afirst digital image presented to the user's left eye and a seconddigital image presented to the user's right eye based on the user'sperceived focal distance and the content type. Perceived focal distance(also referred to as the “viewing distance”) refers to the imagineddistance between the user and the user's focal point within the virtualor augmented reality environment. The adjustment may take place in realtime based on measurements and analysis of the eye movement andperceived focal distance of the user. The perceived focal distance ofthe user can be measured in a variety of ways, including an ultrasonicranging system, eye and/or head position tracking, etc.

The HMD system can adjust the overlap by decreasing or increasing thenumber of pixels viewed by the user's left eye and/or right eye. Forexample, if pixel density increases at a central focal point (i.e.,between left and right eyes), the overlap is likely to increase. Theoverlap adjustment methods described herein can be performedautomatically by the HMD or manually (e.g., by an administratorassociated with the HMD or a programmer/developer associated with thedigital content). Automatic adjustments could also be made by anelectronics module (e.g., electronics module 218 of FIG. 2A) thatdetects the display mode (i.e., binocular or monocular), the type (i.e.,2D or 3D content), the application (e.g., video, gaming, text, document,map), etc., of the digital content presented to the user and makesadjustments accordingly. Manual adjustments, meanwhile, could be made bythe user using a tactile input sensor, providing a vocal command, etc.

As described above, the overlap presented by the HMD at a given point intime can be based on the perceived focal distance of the user. That is,the imagined distance between the user and the focal point of the user'seyes. In some embodiments, the amount of overlap is predetermined basedon the measured focal distance. For example, an embodiment may supportthree viewing distances: close-range (45-55 cm); medium-range (70 cm-1m); and long-range (>3 m). In some embodiments, the user may have theoption of selecting one of the predetermined overlaps. The user'sselection, which may provide a more comfortable viewing experience, canbe based on the content type, display mode, application type, or somecombination thereof. The user may be able to change the overlap in avariety of ways. For example, in some embodiments the user can use avocal command as an input, while in other embodiments the user canperform a gesture using a hand, arm, leg, etc. The user could, forinstance, present the palm of her hand to a sensor (e.g., camera) inorder to select the close-range viewing distance and a first forlong-range viewing distance. One skilled in the art will recognize thatmany vocal commands and gestures are possible as input commands. The HMDsystem can include one or more sensors (e.g., sensors 214 of FIGS. 2A-B)that are adapted to recognize the gestures and/or vocal commands of theuser. Examples of possible gestures include tilting the head down forclose-range view, tilting head up for long-range view, extending arm forlong-range view, etc.

The overlap modification can also trigger a change in how digitalcontent is presented to the user. For example, brightness, contrast, andinput mode may change when a vocal command or gesture triggers a commandinput. Similarly, modifications to the overlap may cause an increase inperipheral presentation while reducing or eliminating centralpresentation. A modification may also occur when the user selects adifferent viewing mode. In some embodiments, the HMD modifies a userinterface or makes other visual adjustments based on the user's eyemovement and position (e.g., interpupillary distance). Additionally oralternatively, overlap modifications could be triggered by the user'smotion (e.g., walking, running) and/or location (e.g., when in a movingvehicle or restaurant).

Touch-Based Content Management

FIG. 5 is an inside view of an HMD 500 that includes configurable icons518. The HMD 500 includes a bridge 508, optical display surfaces 512Land 512R (collectively “optical display surface 512”), one or moretactile sensors 516, and icons 518. The bridge 508 may be supported bythe user's nose to assist in stabilizing the HMD 500 and, as describedabove, the optical display surface 512 may display digital content to auser using a variety of different configurations. In some embodiments,the HMD 500 displays one or more icons 518 to the user. The icons 518may allow the user to further interact with the local environment (e.g.,by interacting with digital content related to the local environment),access information pertaining to the local environment (e.g., socialmedia comments, restaurant reviews, directions), or receivenotifications (e.g., personal email or text messages).

One or more tactile sensors 516 can be coupled to the frame that allowthe user to modify what digital content is displayed on the opticaldisplay surface 512 or interact with the displayed content (e.g., icons518). The one or more tactile sensors 516 can be integrated into theframe or coupled to the frame depending on the application. For example,the tactile sensor(s) 516 may be integrated into the frame if the HMDtakes the same form as conventional eyeglasses. The tactile sensor(s)516 may include piezoresistive sensors, piezoelectric sensors,capacitive sensors, electroresistive sensors, etc. The tactile sensor(s)516 may also be implemented in a variety of ways, such as buttons,switches, touch screens, etc. In some embodiments, the tactile sensor(s)516 include pressure-sensitive multi-touch sensors (i.e., 3D Touch) thatallow the user to convey specific commands (e.g., slight pressureresults in minor realignment of the digital content and more significantpressure causes major realignment of the digital content).

In some embodiments, the one or more tactile sensors 516 are used tomodify the overlap displayed on the optical display surface 512. Forexample, pressure applied to a tactile sensor embedded within the framenear the user's left eye may modify the overlap for the user's left eye,while pressure applied to a tactile sensor embedded within the framenear the user's right eye may modify the overlap for the user's righteye. In some embodiments, the user is able to apply pressure to bothsides of the frame to modify the overlap of both eyes. Generally, thetactile sensor(s) 516 are coupled to the outer edge of the frame and/oroptical display surface 512. However, the tactile sensor(s) 516 couldalso be coupled inside the frame and/or optical display surface 512nearer to the user's eyes. In some embodiments, the tactile sensor(s)516 are embedded within the frame and/or optical display surface 512.

The HMD 500 may utilize gestures as input commands or as a means tomodify the user interface. Gestures may include, for example, pushingthe bridge 508 of the frame, applying pressure to the frame with one orboth hands, and pushing the frame upward or laterally. The gestures canbe used to modify the user interface or trigger a command input (e.g.,select, zoom, modify overlap, modify resolution) for the HMD. Thesegestures could be used in both monocular and binocular HMD systemsregardless of whether the HMD systems are configured to presentaugmented or virtual reality content.

The digital content displayed to the user may change depending onwhether the HMD 500 is presenting augmented or virtual reality content.For example, an HMD 500 presenting augmented reality content may displaydigital imagery only in the user's peripheral area. The layout of thedigital imagery may be a single icon, a column of icons, a collapsibletable of icons, etc. In some embodiments, peripheral icons are designed(e.g., size, shape, brightness) such that the icons are easilydetectable, but not clearly viewable in the user's peripheral area. Forexample, the icons 518 of FIG. 5 are located in the user's peripheralarea. The brightness of these icons 518 could naturally be set to a lowlevel and may increase if new information or an update becomes availableto the user.

By applying pressure next to (or directly on top of) the peripheral icon518, the user can interact with the icon 518. The intensity level andduration of the pressure may affect which action(s) are performed by theHMD (e.g., select the icon, push content into the user's visual field,hide the icon). Similarly, the force and duration of the pressure maytrigger different input commands (e.g., zoom, modify overlap, modifyresolution). For example, if the user applies pressure to a tactilesensor 516 next to the icon 518, the icon may move from the peripherytowards the center of the optical display surface 512. If the usermaintains contact with the tactile sensor 516, the icon 518 or itscontents may remain stationary for the user to observe. If the userdiscontinues contact with the tactile sensor 516, the icon 518 mayreturn to its original peripheral position. In some embodiments, theicon 518 returns to its original peripheral position when the userapplies pressure to a tactile sensor 516 on the opposite side of theoptical display surface 512. Generally, the HMD 500 can also distinguishbetween a single pressure point (e.g., one finger, one button) andmultiple pressure points (e.g., multiple fingers, multiple buttons),which allows the HMD to recognize a greater number of user interface andinput commands.

The icons 518 can notify the user in various ways, including an increasein brightness, movement of an existing icon, appearance by a new icon,etc. In some embodiments, the icon 518 moves into the middle of theuser's field of view in response to a first pressure to a tactile sensor516. If the user applies a second pressure to the tactile sensor 516,the icon 518 may return to its original peripheral location. In variousembodiments, the effect of one or more user interactions (e.g., length,amount, and/or location of pressure applied) can be predefined and/orcustomized by the user (e.g., using an application, software program, orweb-based interface associated with the HMD).

Customization may allow the user to modify the HMD 500 in order tobetter accommodate her viewing needs and improve the effectiveness ofthe HMD 500. The user may be able to customize various features,including the icon presentation, user interface layout, number of iconsdisplayed, information to be displayed by the icons, etc. In someembodiments, different parts of the frame, bridge 508, and/or opticaldisplay surface 512 can be customized to trigger various user interfacemodifications and/or command inputs. For example, the bridge 508 may beconfigured to power on/off the HMD 500 when pressure is applied by theuser.

In some instances, the virtual or augmented reality content to bedisplayed to the user requires an overlap that the HMD 500 is unable toprovide (e.g., exceeds system capabilities). In order to avoid visualdiscomfort, the HMD 500 can switch from a binocular viewing mode to amonocular viewing mode (or vice versa) to more comfortably present thedigital content. For example, the HMD 500 may elect to present contentto only one eye by displaying the digital content on optical displaysurface 512L or optical display surface 512R. The amount of pressureapplied by the user to a tactile sensor 516 may also be used foradjusting the viewing angle, resolution, etc., of the digital contentdisplayed in the monocular or binocular viewing mode.

Lighting-Based Modification

In various embodiments, the HMD system 500 also includes a sensor 514coupled to the frame and/or the optical display surface 512 to monitorvarious aspects of the user's local real world environment. The sensor514 may be a camera configured to capture the user's interactions withthe local environment, a light sensor configured to track illuminancelevels within the local environment, an audio sensor configured toidentify vocal commands, etc. In some embodiments, a plurality ofsensors 514 are employed that serve as a light detection system, whichmeasures the ambient light in the environment and/or any illuminationdirected towards the plurality of sensors 514. Based on the measuredilluminance levels, the user interface (e.g., icon arrangement, size)and/or the input management of the HMD 500 can be modified. For example,the size of the icons 518 could be increased in low-light environments.These larger icons may require less precise finger tracking and inputcommand detection (which may be necessary in certain lightingenvironments that prevent certain detection systems, such as infrared,from operating properly). Similarly, the brightness, color, background,etc., of the user interface and any digital content displayed on theoptical display surface 512 can be modified in response to the measuredilluminance levels.

In some embodiments, the HMD 500 alerts the user to use alternativeinput command gestures based on the illuminance levels measured by thelight detection system. The HMD system 500 can, for example, be adaptedto warn the user of potential tracking and gesture detection issues inlow-light environments. The HMD system 500 could also be adapted toswitch to an alternative command input and/or user interface systembased on the measured illuminance level. For example, in bright lightenvironments, the HMD system 500 can limit the number of trackablegestures and allow only a subset of predefined gestures.

In various embodiments, the measured illuminance level can affect theoverlap of the digital content presented to the user, the fusionalstabilization level, the content mode (e.g., 2D, 3D) displayed to theuser, the display mode (e.g., binocular, bi-ocular, monocular) of theHMD system 500, or any combination thereof.

FIG. 12 is a flowchart of a process 1200 for modifying an interfaceand/or updating the controls for an input management system as may occurin some embodiments. Conventionally, infrared tracking systems are usedto monitor and identify gestures performed by a user to interact withdigital content presented by an HMD. However, infrared tracking systemsare generally only effective in a small sub-set of lighting environmentsand are largely ineffective in atypical lighting environments, such asbright-light outdoor environments. Various embodiments described hereinovercome these problems by modifying the user interface and/or inputmanagement system of the HMD based on the illuminance levels of thelocal environment.

At step 1202, the HMD detects the ambient illuminance (e.g., usingsensors coupled to the frame of the HMD) of the local environment. Atstep 1204, the user interface and/or input management controls of theHMD are modified based on the ambient illuminance levels. For example,icons that compose the user interface may be enlarged to account forless accurate gesture identification. The controls of the HMD could alsobe modified if the HMD determines a first set of management controls isno longer appropriate. For example, the HMD may require that commands beprovided via audible comments or touch-based commands (e.g., usingbuttons on the frame of the HMD), rather than gestures. At step 1206,the updated user interface is presented to the user and/or the updatedinput management system is implemented by the HMD.

Animated Augmented Content

A major challenge for augmented reality systems is correctsuperimposition of digital content onto a 3D real world environment,particularly in response to constantly changing fixation distances.Superimposing digital content onto 3D real world content requiresaccurate measuring and analyzing of the actual distance of the realworld object, the perceived distance of the digital content, cameracalibration, eye gaze movement, and head positioning (among otherfactors). The HMDs described herein can use animated digital content tosimulate a change of distance in augmented reality systems. Variousproperties (e.g., convergence, brightness) of the animated digitalcontent can be modified to more accurately imitate the change in theuser's focal distance. For example, at close ranges the digital contentmay increase in size and appear closer to the eyes of the user, therebyincreasing the convergence and demand. At longer ranges the digitalcontent may decrease in size and appear farther away from the eyes ofthe user, thereby reducing the convergence and demand.

In some embodiments, the animation of the digital content can betriggered and/or controlled by the user. For example, the user maypresent a hand gesture, apply pressure to the frame, voice a command,tilt her head down, etc., in order to view the animated digital contentat a closer (or farther) distance. The user may also present a handgesture, voice a command, etc., to adjust and/or superimpose the digitalcontent on a specific location relative to the 3D real world environment(e.g., on top of a targeted object). In some embodiments, the user isable to move and/or modify a plurality of digital images, or distinctparts of a single digital image, simultaneously in order to provide moreaccurate depth perception.

Vision Stabilization

FIGS. 6A-C depict how visual stabilizers can be used to readily andeffectively merge distinct images into a single composite image. Oncevisual stabilizers have been generated, they can be integrated into thedigital content to be shown to the user (e.g., by superimposing thevisual stabilizers on top of the digital content). Visual stabilizerscan be used to improve a variety of issues that plague users of HMDs,including the fatigue experienced because there is no change in focaldistance. The visual presentation system 600 presents one or more visualstabilizers to each eye that are merged together in binocular view tocreate a strong fusional system. The strong fusional system allows theuser to merge and maintain the two separate images that have a smallamount of overlap without visual discomfort. Moreover, a strong fusionalsystem allows visual discomfort to be lessened (or eliminated entirely)by reducing the negative effects, such as eye strain, typicallyassociated with viewing two different digital images simultaneously(e.g., by more accurately aligning the images). When digital content ispresented by an HMD that has a large amount of overlap, the fusionalsystem need not be as strong (e.g., because so many visual elements areshared between the two images).

The visual presentation system 600 can include a plurality of visualstabilizers that are added to the digital content presented to each eyeof the user. The visual stabilizers provide visual cues that help theuser merge or “lock” the images together. Generally, if the digitalcontent presented to each of the user's eyes is similar or identical,fewer visual stabilizers are used because the digital content can bemore easily merged together by the user. For example, the visualstabilizers may be a first outer frame 630 and a first inner focal point632 presented in one image, and a second outer frame 634 and a secondinner focal point 636 presented in another image. These visualstabilizers could be added to the digital content during development oradded to pre-existing content (e.g., web pages, text messages, games,maps, videos, pictures) capable of being displayed on an optical displaysurface. In some embodiments, the visual stabilizers are distinctgeometric shapes that are integrated into the digital content.Additionally or alternatively, elements already or naturally present inthe digital content (e.g., icons, elements of a user interface) may beused as visual stabilizers. Such visual stabilizers may beindistinguishable from the rest of the digital content by the user.

As shown in FIG. 6A, the plurality of visual stabilizers can be mergedtogether in binocular view to create a single fused image in which thevisual elements are properly aligned when viewed to the user. The fusedimage may, for example, include a fused outer frame 638 created bycombining the first outer frame 630 and second outer frame 634, and afused inner focal point 640 created by combining the first inner focalpoint 632 and second inner focal point 636. Although the first outerframe 630 and second outer frame 634 are depicted as portions of aframe, they could also each be a complete frame that overlay one anotherwhen viewed simultaneously. Partial shapes may be used to help identifyinstances of eye fatigue and/or be performed as part of ananti-suppression technique (e.g., performed on the non-dominant eye).The visual stabilizers can be located in the peripheral areas of thedigital image or in a more central location nearer the user's focalpoint. Visual stabilizers that are easily visible (e.g., largergeometric shapes, thicker frames, brighter coloration) are oftenpreferred because the user is able to more readily recognize and mergethe visual stabilizers. However, there may be instances, such as videogames or movies, where more inconspicuous visual stabilizers arepreferable.

In some embodiments, the plurality of visual stabilizers fuse to ashared location. For example, the first inner focal point 632 and thesecond inner focal point 636 may converge on the same icon as shown inFIG. 6A. In other embodiments, the plurality of visual stabilizers maybe mapped to different locations. Visual stabilizers that map todifferent locations (e.g., peripheral areas of the user's vision) may bepreferred when the user suffers from suppression-related issues. Forexample, the first inner focal point 632 and the second inner focalpoint 636 of FIG. 6B are placed in distinct locations of the digitalcontent. The visual stabilizers could also affect the amount of overlappresented by the HMD and/or vary based on the amount of overlap shown bythe HMD.

Visual stabilizers can be integrated into digital augmented or virtualreality content at various stages of development. For example, visualstabilizers may be integrated while digital content is being createdthrough the use of software development kits (SDKs). The SDK can includeone or more application program interfaces (APIs) for adding visualstabilizers to digital content. The APIs may include a number offunction modules for implementing various interface functions (e.g.,overlap variation, user interface modification). Pre-coded instructionsfor these various interface functions can also be provided to implementthese interface functions in combination with pre-existing digitalcontent.

As described above, visual stabilizers can be used with augmentedreality content, virtual reality content, and mixed (i.e., somecombination of augmented and virtual reality) content. In someembodiments the visual stabilizers are placed in retinal correspondenceareas, while in other embodiments the visual stabilizers are placed inretinal non-correspondence areas. Retinal correspondence areas are thoseareas that include elements focused on by the user's two eyes that sharea common subjective visual direction (i.e., a common visual direction).

Various properties of the visual stabilizers may change based on thecharacteristics of the viewing experience. These properties include, butare not limited to, size, number, shape, position, color, brightness,and contrast. For example, the visual stabilizers may typically be acircle, an ellipse, a triangle, any other quadrilateral, a line (orcombination of lines), etc. Meanwhile, the viewing characteristics thatprompt the change could include the current size of the overlap shown tothe user, whether the digital content is augmented or virtual realitycontent, the type of digital content (e.g., movies, video games), thetype of optics used in the HMD, the age of the user (e.g., older usersmay require additional visual stabilizers), whether the user has anyknown eye problems, whether the user requires corrective glasses orcontact lenses, the time of day (e.g., fatigue increases later in theday), ambient light levels (e.g., brighter visual stabilizers in outdoorviewing environments), etc. The visual stabilizers may also be temporary(e.g., only used to initially align the images) or permanent. Visualstabilizers could be added as (or after) the eyes become tired. Forexample, visual stabilizers may be added to digital content after acertain time period (e.g., additional visual stabilizers every 30minutes) or after receiving a user input (e.g., user interacts with theHMD and specifies that she is experiencing eye fatigue). In someembodiments, the visual stabilizers are animated elements of the digitalcontent.

In embodiments having a plurality of visual stabilizers, the visualstabilizers may be related and could vary based on the content type,application, mode (e.g., 2D or 3D), local environment, user movement,position/orientation of the user's head, position of the user's eyes,etc. As shown in FIGS. 6A-C, the visual stabilizers are placed aroundthe digital content as visual frames in some embodiments. For example, afused outer frame 638 can form an outer border that surrounds thedigital content. A visual frame may be a complete or partial segment ofany geometric shape (e.g., rectangle, circle, line) that surrounds thedigital content displayed to the user. As the user views the displayedcontent with both eyes, the visual frames merge, thereby creating afully fused and stable visual frame. The properties of the visual frame,such as size, thickness, and length, can vary based on a number offactors, including, but not limited to, overlap size, perceiveddistance, and application. In some embodiments, the digital content ismodified (e.g., shifted left/right or up/down) after determining thespatial relationship between the visual stabilizers does not match anexpected relationship (e.g., arrangement). The modification could alsobe a change in orientation or position of the digital content, totalpixel count, the overlap of the digital images, etc.

Peripheral visual stabilizer can be used for presenting a plurality ofdifferent digital contents in the peripheral region of each eye of auser. For example, FIG. 6C shows a presentation of two differentcontents in the user's peripheral vision with no overlap.

One or more properties of the visual stabilizer can also be changedbefore or during use (i.e., viewing of the digital content). Forexample, a frame (e.g., fused outer frame 638) could be set to apredetermined width for each of a plurality of users, such that digitalcontent viewed by a first user implements a first width, digital contentviewed by a second user implements a second width, etc. In someembodiments, properties of the visual stabilizer are modified while theuser is viewing the digital content. For example, the width of a frame(e.g., fused outer frame 638) may increase proportional to the durationof use. Similarly, the brightness of one or more visual stabilizers mayincrease proportional to the duration of use or the amount of ambientillumination. In some embodiments, additional visual stabilizers areadded while the user is viewing the content. For example, additionalclouds, trees, etc., may be added to the digital content (e.g., 3Dvirtual environment) over time to decrease the viewing fatigueexperienced by the user, which typically affects the user's ability toproperly fuse the two images together (i.e., the user's fusionalsystem).

An HMD system may be configured to implement visual stabilizers that arepersonalized for the user. The personalization can be based on theapplication, content type, length of use, a user's visual impairment(e.g., myopic, hyperopic, diplopia, color blindness), or any of theother viewing characteristics described above. For example, a pluralityof visual stabilizers (e.g., fused outer frame 638, fused inner focalpoint 640) having high resolution and strong contrast might be shown toa first user who has pre-existing visual impairments, while a singlevisual stabilizer (e.g., fused outer frame 638) might be shown to asecond user who has no visual impairments. The single visual stabilizermay be one of the plurality of visual stabilizers shown to the firstuser or another distinct visual stabilizer. Similarly, a plurality ofvisual stabilizers shown to a first user can be different than thoseshown to a second user. For example, a first series of visualstabilizers (e.g., fused outer frame 638, fused inner focal point 640)could be shown to a first user, and a different second series of visualstabilizers (e.g., trees, clouds) could be shown to a second user.Animated and/or temporary visual stabilizers that are only viewable fora limited amount of time could also be personalized for the user (e.g.,visual stabilizers may be presented to users who have visual impairmentsor who have used the HMD for an extended period for a longer period).The personalization can be implemented manually (i.e., directed by userinput) or automatically based on content type, length of use, etc.

High-Resolution Perception

FIG. 7 is a diagram illustration of a user's high resolution and lowresolution viewing fields. As further described below, some embodimentsdescribed herein improve the perception and image stability of HMDs byincreasing resolution in a particular area of the digital contentdisplayed to the user 702. When the user 702 trains her eyes on a focalpoint 740 in a real-world environment, a high resolution field of view742 and a low resolution field of view 744 are created. The highresolution field of view 742 is limited in area and generally isapproximately 2 degrees. Various embodiments herein emulate thesenatural fields of view by increasing resolution in certain areas in adigital image, while decreasing resolution in other areas.

FIG. 8 depicts a user's high resolution viewing fields 842 and lowresolution viewing fields 844 in a single instance. As described above,when the user trains her eyes on a focal point 840, a high resolutionfield of view 842 and a low resolution field of view 844 are created. Ifthe user trains her eyes on a plurality of focal points in succession(e.g., focal points A and B), a low resolution field of view 844 will becreated between those focal points.

FIG. 9 is an illustration of high resolution and low resolution viewingareas in a digital image presented by an HMD. The digital image of FIG.9 includes a focal point 940 that is the subject of a user's gaze, ahigh resolution area 942 surrounding the focal point 940, and a lowresolution area 944 that encompasses the remainder of the digital image.The HMD system can vary the resolution of the digital content displayedto a user based on various criteria, such as the speed and direction ofthe user's head and/or eye movements. In some embodiments, the HMDdisplays digital content having a higher resolution in an area 942 nearor around the focal point 940 of the user's eyes. The remaining area ofthe digital content may be displayed in a lower resolution. Thesetechniques more accurately mimic the resolution naturally experienced bythe user's eyes (i.e., high resolution only where the user's gaze isactually focused). In some embodiments, the size of the high resolutionarea 942 varies based on the speed of the user's head or eye movements.

Adjustments to the resolution of the digital content can be made by anelectronics module (e.g., electronics module 218 of FIGS. 2A-B). Theelectronics module may employ predictive algorithms (or other forms ofmachine learning) to better predict the adjustments likely to benecessary for particular users and/or types of digital content. Forexample, in some embodiments these algorithms monitor the user's headmovement to predict which area(s) of the digital content should have ahigh resolution at a certain time, while in other embodiments thesealgorithms analyze the digital content itself to determine where a useris likely to look (e.g., a particular location on the display whenplaying a video game) at a certain time. The adjustments generally takeplace in real time.

In some embodiments, the resolution of the peripheral area increases ordecreases relative to the speed and direction of the user's head or eyemovements. When the user's eyes move from a first focal point to asecond focal point (e.g., focal points A and B of FIG. 8), theresolution of the digital content between the two focal points may bereduced. The reduction can be proportional to the speed of the user'shead or eye movements. For example, the resolution may decrease when thespeed of user's head and/or eye movement increases. The reduction canalso be based on the direction of the user's head or eye movement (e.g.,resolution decreases when the user's head is tilted downward andincreases when the user's head is tilted upward). Similarly, theresolution may increase as the speed of the user's head and/or eyemovement decreases, which may indicate the user is attempting to focuson an element of the digital content. In some embodiments, the HMDsystem is configured to display the maximum resolution across theentirety of the digital content (or some portion thereof) when there isno movement by the user's head and/or eyes. The high resolution area 942preferably moves within the bounds of the digital content according towhere the user's gaze is focused. Movement of the high resolution area942 can depend on tracking of speed and direction of the user's head oreye movement by the HMD system. However, in some embodiments the highresolution area 942 is fixed in a predetermined location.

FIG. 11 is a flowchart of a process 1100 for adjusting displayresolution by the HMD as may occur in some embodiments. In someembodiments, the HMD is adapted to modify the resolution of digitalcontent presented to the user in real-time. At step 1102, the HMDdetermines the focal point of the user's gaze. This can be done byidentifying certain gestures (e.g., pointing) or by tracking the speedand direction of the user's eye and/or head movement, as shown at step1104. At step 1106, the resolution of the digital content is adjustedaround the focal point. Typically, the HMD will increase the resolutionin an area around or near the focal point, and decrease the resolutioneverywhere else in the digital image.

Multi-Distance Optical System

FIG. 10 is a diagram illustration of a method for displaying digitalcontent over a range of focal distances. As described above, some HMDsinclude a first optic 1052 configured to present digital content up to apredetermined distance away, and a second optic 1050 configured topresent any digital content whose distance exceeds the predetermineddistance. For example, the first optic 1052 may be configured to presentcontent for close-range viewing, while the second optic 1050 may beconfigured to present content for long-range viewing. Such an HMD canalso include an electronics module 1018 that includes or iscommunicatively coupled to a visual balancing module 1054. The visualbalancing module 1054 can use a fusional stabilizer system to balancethe two distinct images displayed on the first optic 1052 and the secondoptic 1050. The visual balancing module 1054 can also output a fusedcomposite image to be shown to a user on, for example, an opticaldisplay surface of an HMD.

Computer System

FIG. 15 illustrates a block diagram of a computer system that may beused to implement certain features of some of the embodiments describedherein. The computer system 1500 may be a server computer, a clientcomputer, a personal computer (PC), a user device, a tablet PC, a laptopcomputer, a personal digital assistant (PDA), a cellular telephone, aniPhone, an iPad, a Blackberry, a processor, a telephone, a webappliance, a network router, switch or bridge, a console, a hand-heldconsole, a (hand-held) gaming device, a music player, any portable,mobile, hand-held device, wearable device, or any machine capable ofexecuting a set of instructions, sequential or otherwise, that specifyactions to be taken by that machine.

The computing system 1500 may include one or more central processingunits (“processors”) 1502, memory 1504, a communication device 1506, andan input/output device 1508 (e.g., keyboard and pointing devices, touchdevices, display devices) that are connected to an interconnect 1510.

The interconnect 1510 is illustrated as an abstraction that representsany one or more separate physical buses, point-to-point connections, orboth connected by appropriate bridges, adapters, or controllers. Theinterconnect, therefore, may include, for example a system bus, aperipheral component interconnect (PCI) bus or PCI-Express bus, aHyperTransport or industry standard architecture (ISA) bus, a smallcomputer system interface (SCSI) bus, a universal serial bus (USB), IIC(12C) bus, or an Institute of Electrical and Electronics Engineers(IEEE) standard 1394 bus, also referred to as Firewire.

The memory 1504 is any computer-readable storage media that storesinstructions that implement at least portions of the various embodimentsdescribed herein. In addition, the data structures and messagestructures may be stored or transmitted via a data transmission medium(e.g., a signal on a communications link). Various communications linksmay be used, such as the Internet, a local area network, a wide areanetwork, or a point-to-point dial-up connection. Thus, computer readablemedia can include computer-readable storage media (e.g., non-transitorymedia) and computer-readable transmission media.

The instructions stored in memory 1504 can be implemented as softwareand/or firmware to program one or more processors 1502 to carry out theactions described above. In some embodiments, such software or firmwaremay be initially provided to the processor 1502 by downloading it from aremote system through the communication device 1506 (e.g., Ethernetadapter, cable modem, Wi-Fi adapter, cellular transceiver, Bluetoothtransceiver).

The various embodiments introduced herein can be implemented by, forexample, programmable circuitry (e.g., one or more microprocessors),programmed with software and/or firmware, entirely in special-purposehardwired (i.e., non-programmable, circuitry), or in a combination ofsuch forms. Special-purpose hardwired circuitry may be in the form of,for example, one or more ASICs, PLDs, FPGAs, etc.

The computing system 1500 may be communicatively coupled to theelectronics module (e.g., electronics module 218 of FIG. 2A-B) or theHMD. The digital content modification, image processing, and/or dataanalyzing techniques described herein can be accomplished solely by thecomputing system 1500 or shared between the computing system 1500 andthe electronics module. In some embodiments the computing system isphysically coupled to the HMD, while in other embodiments the computingsystem may communicate with the HMD and/or electronics module wirelessly(e.g., via Bluetooth or Wi-Fi).

In addition to the above mentioned examples, various other modificationsand alterations of the invention may be made without departing from theinvention. The language used in the Detailed Description has beenprincipally selected for readability and instructional purposes, and itmay not have been selected to delineate or circumscribe the inventivesubject matter. Accordingly, the above Detailed Description is not to beconsidered as limiting and the appended claims are to be interpreted asencompassing the true spirit and the entire scope of the invention.

What is claimed is:
 1. A system for regionally modifying resolution ofdigital content to emulate real-world fields of view, the systemcomprising: a processor communicatively coupled to a head-mounted deviceoperated by a user and operable to execute instructions stored in amemory; and the memory, which includes specific instructions forregionally modifying the resolution of digital content displayed by thehead-mounted device based on a location of a focal point of the user'sgaze, wherein execution of the specific instructions causes theprocessor to: cause digital content to be displayed by the head-mounteddevice; identify a location of the focal point at a first point in time;identify a first area of the digital content that includes the focalpoint at the first point in time; increase resolution of the first areaof the digital content; identify a second area of the digital contentthat is likely to include the focal point at a second point in time;increase resolution of the second area of the digital content; anddecrease resolution of a third area of the digital content that islocated between the first and second areas of the digital content,wherein the resolutions of the first, second, and third areas aremodified substantially simultaneously.
 2. The system of claim 1, whereinexecution of the specific instructions further causes the processor to:continuously monitor the focal point in real time; determine whether thefocal point has changed; and responsive to a determination that thefocal point has changed, modifying a size or a position of the firstarea of the digital content.
 3. The system of claim 2, wherein saidcontinuously monitoring includes monitoring movement of the user's eyes,head, or both.
 4. The system of claim 1, wherein the resolution of thethird area is decreased proportional to eye movement speed, headmovement speed, or some combination thereof.
 5. The system of claim 1,wherein the third area is the remainder of the digital content that isnot included in the first area or the second area.
 6. The system ofclaim 1, wherein the processor uses predictive algorithms to identifythe second area of the digital content that is likely to include thefocal point at the second point in time.
 7. The system of claim 6,wherein the predictive algorithms consider speed and direction of theuser's head movement, a digital element of the digital content, or both.8. A head-mounted device comprising: a supportive frame; an opticaldisplay surface coupled to the supportive frame; an image display systemconfigured to cause display of digital content on the optical displaysurface; a first sensor configured to continuously track a focal pointof a user's gaze in real time; a second sensor configured tocontinuously monitor eye movement speed or head movement speed; aprocessor, communicatively coupled to the image display system,configured to execute instructions stored in a memory; and the memory,which includes specific instructions for regionally modifying resolutionof the digital content displayed on the optical display surface, whereinexecution of the specific instructions causes the processor to: cause afirst digital image to be displayed on the optical display surface;identify a location of the focal point at a first point in time;identify a first area of the first digital image that includes the focalpoint at the first point in time; increase resolution of the first areaof the first digital image; identify a second area of the first digitalimage that does not include the focal point at the first point in time;and decrease resolution of the second area of the first digital image,wherein the decrease in resolution of the second area of the firstdigital image is proportional to the eye movement speed or the headmovement speed, and wherein the resolutions of the first and second areaare modified substantially simultaneously.
 9. The head-mounted device ofclaim 8, wherein the changes in resolution of the first and second areasare performed for the purpose of mimicking a high-resolution viewingfield and a low-resolution viewing field naturally experienced by theuser when viewing a real-world environment.
 10. The head-mounted deviceof claim 8, wherein the second area includes a series of sections, andwherein the resolution of each section is decreased proportional to thedistance between the section and the focal point of the user's gaze. 11.The head-mounted device of claim 8, further comprising: a third sensorconfigured to detect gestures performed by the user, wherein saididentifying the location of the focal point at the first point in timeis based at least in part on a gesture performed by the user.
 12. Thehead-mounted device of claim 8, wherein execution of the specificinstructions further causes the processor to: identify a candidate areaof a second digital image that is likely to become the subject of theuser's gaze at a second point in time using a predictive algorithm,wherein the candidate area is part of the second area of the firstdigital image at the first point in time.
 13. The head-mounted device ofclaim 12, wherein machine learning techniques are employed toperiodically improve the predictive algorithm.
 14. The head-mounteddevice of claim 12, wherein the predictive algorithm considers as inputdirection of the user's head movement, speed of the user's headmovement, or both.
 15. The head-mounted device of claim 12, wherein thepredictive algorithm considers as input a digital element of the seconddigital image to be displayed at the second point in time and that isdetermined to likely become the subject of the user's gaze.
 16. A methodfor regionally modifying resolution of digital content that is presentedto a user of a head-mounted device, the method comprising: causing ahead-mounted device to display digital content for viewing by a user;determining where a focal point of the user's gaze is located at a firstpoint in time; identifying a first area of an optical display surfacethat includes the focal point at the first point in time; increasingresolution of a first digital image within the first area; identifying asecond area of the optical display surface that is determined to likelyinclude the focal point at a second point in time at which a seconddigital image is shown; determining that the focal point has begun tomove toward the second area of the optical display surface; increasingresolution of the second digital image within the second area;identifying a third area of the optical display surface located betweenthe first area and the second area; and decreasing resolution of thefirst and second digital images within the third area; wherein theresolutions of the first and second digital images are modifiedsubstantially simultaneously as the focal point moves toward the secondarea of the optical display surface.
 17. The method of claim 16, whereinthe third area is the remainder of the optical display surface notincluded in the first area and the second area.
 18. The method of claim16, wherein the third area is composed of a series of sections that eachinclude a plurality of pixels, and wherein the resolution of eachsection is decreased proportional to the distance between the sectionand the first area or the second area.
 19. The method of claim 16,wherein the second area of the optical display surface is located in afixed, predetermined location.
 20. The method of claim 19, wherein thefixed, predetermined location corresponds to a feature of the digitalcontent that is always or nearly always viewable by the user.